Medical devices, systems, and methods using eye gaze tracking for stereo viewer

ABSTRACT

An eye tracking system may comprise an image display configured to display a stereo image of a surgical field to a user, a right eye tracker configured to measure data about a first gaze point of a right eye of the user, and a left eye tracker configured to measure data about a second gaze point of a left eye of the user. The eye tracking system may also comprise at least one processor configured to process the data about the first gaze point and the second gaze point to determine a viewing location in the displayed stereo image at which a three-dimensional gaze point of the user is directed.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/175,295, filed Feb. 12, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/547,869, now U.S. Pat. No. 10,965,933, filedAug. 22, 2019, which is a divisional of U.S. patent application Ser. No.15/126,151, now U.S. Pat. No. 10,432,922, filed Sep. 14, 2016, which isthe U.S. National Phase of International Patent Application No.PCT/US2015/021315, filed Mar. 18, 2015, which designated the U.S. andclaims priority to and the benefit of U.S. Provisional PatentApplication No. 61/955,334, entitled “Medical Devices, Systems, andMethods Integrating Eye Gaze Tracking for Stereo Viewer,” filed Mar. 19,2014, all of which are incorporated by reference herein in theirentirety.

BACKGROUND

Surgical procedures can be performed using a teleoperational medicalsystem in a minimally invasive manner. The benefits of a minimallyinvasive surgery are well known and include less patient trauma, lessblood loss, and faster recovery times when compared to traditional, openincision surgery. In addition, the use of a teleoperational medicalsystem, such as the DA VINCI® Surgical System commercialized byIntuitive Surgical, Inc., Sunnyvale, Calif., is known. Suchteleoperational medical systems may allow a surgeon to operate withintuitive control and increased precision when compared to manualminimally invasive surgeries.

A teleoperational medical system may include one or more instrumentsthat are coupled to one or more robotic arms. If the system is used toperform minimally invasive surgery, the instruments may access thesurgical area through one or more small openings in the patient, such assmall incisions or natural orifices, such as, for example, the mouth,urethra, or anus. In some cases, rather than having the instrument(s)directly inserted through the opening(s), a cannula or other guideelement can be inserted into each opening and the instrument can beinserted through the cannula to access the surgical area. An imagingtool such as an endoscope can be used to view the surgical area, and theimage captured by the imaging tool can be displayed on an image displayto be viewed by the surgeon during a surgery.

It is desirable to provide teleoperational medical systems that caneffectively and accurately employ eye gaze tracking for variousapplications during minimally invasive medical procedures. The systemsand methods disclosed herein overcome one or more of the deficiencies ofthe prior art.

SUMMARY

In one exemplary aspect, the present disclosure is directed to an eyetracking system comprising an image display configured to display animage of a surgical field to a user. The image display is configured toemit a light in first wavelength range. The system further comprises aright eye tracker configured to emit light in a second wavelength rangeand to measure data about a first gaze point of a right eye of the user.The system further comprises a left eye tracker configured to emit lightin the second wavelength range and to measure data about a second gazepoint of a left eye of the user. The system further comprises an opticalassembly disposed between the image display and the right and left eyesof user. The optical assembly is configured to direct the light of thefirst and second wavelength ranges such that the first and secondwavelengths share at least a portion of a left optical path between lefteye and the image display and share at least a portion of a rightoptical path between the right eye and the image display, without theright and left eye trackers being visible to the user. The systemfurther comprises at least one processor configured to process the dataabout the first gaze point and the second gaze point to determine aviewing location in the displayed image at which the gaze point of theuser is directed.

In another exemplary aspect, the present disclosure is directed to aneye tracking system comprising an image display configured to display astereo image of a surgical field to a user. The system also comprises atleast one right eye tracker configured to measure data about a firstgaze point of a right eye of the user and at least one left eye trackerconfigured to measure data about a second gaze point of a left eye ofthe user. The system also comprises a right eye light emitter and a lefteye light emitter. The right eye light emitter configured to emit lightof a first wavelength range to the right eye of the user, and the lefteye light emitter configured to emit light of a first wavelength rangeto the left eye of the user. The system also comprises an opticalassembly positioned between the image display and the eyes of the user.The optical assembly comprising a right eye mirror set and a left eyemirror set arranged to provide optical communication between the eyes ofthe user, the eye trackers, and the light emitters. The system alsocomprises at least one processor configured to process the data aboutthe first gaze point and the second gaze point to determine a viewinglocation in the displayed stereo image at which the gaze point of theuser is directed.

These and other embodiments are further discussed below with respect tothe following figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, the 5dimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1A illustrates an exemplary teleoperational medical systemaccording to one embodiment of the present disclosure.

FIGS. 1B, 1C, and 1D illustrate exemplary components of ateleoperational medical system according to various embodiments of thepresent disclosure. In particular, FIG. 1B illustrates a front elevationview of an exemplary teleoperational assembly according to oneembodiment of the present disclosure. FIG. 1C illustrates a frontelevation view of an exemplary operator input system according to oneembodiment of the present disclosure. FIG. 1D illustrates a front viewof an exemplary vision cart component according to one embodiment of thepresent disclosure.

FIG. 2A illustrates a block diagram of the 3D coordinate frames of auser relative to an exemplary image display and a surgical fieldaccording to one embodiment of the present disclosure.

FIG. 2B is a flowchart illustrating an exemplary method of using the eyetracking units to affect the teleoperational system and/or a surgicalinstrument according to one embodiment of the present disclosure.

FIGS. 3A-3D schematically illustrate various embodiments of an eyetracking system of the stereo viewer used by the teleoperational medicalsystem of FIGS. 1A, 1B, and 1C according the present disclosure.

FIG. 4A illustrates a method for determining the surgeon's 3D gaze pointusing the eye tracking system of FIGS. 3A-3D according to one embodimentof the present disclosure.

FIG. 4B is a schematic drawing illustrating an example of eye trackingunits tracking the corneal reflection and the pupil of the surgeon whena predetermined target T is shown on a display during the calibrationprocess according to one embodiment of the present disclosure.

FIG. 4C is a schematic drawing illustrating an example of 2D coordinateframes corresponding to the corneal reflection and the pupillaryposition of the surgeon according to one embodiment of the presentdisclosure.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the disclosed embodiments. However, it willbe obvious to one skilled in the art that the embodiments of thisdisclosure may be practiced without these specific details. In otherinstances well known methods, procedures, components, and circuits havenot been described in detail so as not to unnecessarily obscure aspectsof the embodiments of the disclosure.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein are for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers 30 to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an elongated object.

It will be appreciated that the terms “proximal” and “distal” are usedherein with reference to a clinician manipulating an end of aninstrument extending from the clinician to a surgical site. The term“proximal” refers to the portion of the instrument closer to theclinician, and the term “distal” refers to the portion of the instrumentfurther away from the clinician and closer to the surgical site. Forconciseness and clarity, spatial terms such as “horizontal,” “vertical,”“above,” and “below” may be used herein with respect to the drawings.However, surgical instruments are used in many orientations andpositions, and these terms are not intended to be limiting and absolute.

The present disclosure relates generally to using eye tracking systemsto observe and measure characteristics of a user's eyes (e.g., eye gazetracking) during the use of teleoperational medical systems and/orinstruments used in a variety of medical procedures, including withoutlimitation diagnostic, surgical, and/or therapeutic procedures. Inparticular, in some embodiments, the eye tracking systems disclosedherein rely on the ability to track the accurate location (e.g., 2D or3D location) of the user's eye gaze on a surgical console. In someembodiments, the eye tracking systems may be used to control theteleoperational system by directly operating the system instrumentsand/or by influencing system characteristics to effect system-widechanges. In particular, some embodiments of the present disclosure arerelated to system and instrument control by accurately tracking theoperator's eye gaze on a surgical console while the operator uses ateleoperational medical system during a minimally invasive procedure.

In a teleoperated surgical system, the eye gaze points of the surgeonmay be tracked during a surgery by one or more eye trackers, such asstereo cameras. However, eye gaze tracking may be inaccurate due tovarious factors, including, by way of non-limiting example, changes inthe head position of the user and the separate image displays for eacheye (e.g., creating a stereo view). For example, the pupil's positionand corneal reflection of a surgeon's eye can be determined by acombination of the surgeon's head and eye orientation. In many cases,the head motions of the surgeon, the head pressure of the surgeon uponthe console, and/or image occlusion by the eye trackers during a surgerymay compromise accurate and effective eye gaze tracking usingconventional techniques. Conventionally, eye gaze tracking techniquesrequire an external device disposed out of the teleoperated surgicalsystem to be used during an eye tracking process. For example, theexternal device may be mounted on a pair of glasses that are worn by thesurgeon during a surgery. There is usually a distance and a relativemotion between the external eye tracking device and the surgeon's eyes.Therefore, this kind of external eye tracking device may not only createinconvenience and an uncomfortable feeling to the surgeon, and may alsoaffect the accuracy of the surgeon's operation. Alternatively, theconventional eye gaze tracking device may be located near the eyepieces.This configuration may create interference to the surgeon's vision whenthe surgeon is looking into the eyepieces. For example, the edges of theeye gaze tracking device may appear in the surgeon's vision, which coulddistract the surgeon or compromise his or her view of the surgicalfield. In some embodiments, the eye gaze tracking devices describedherein are configured to enable the eye trackers to share at least partof the same optical path as the displayed image while remaininginvisible to the surgeon.

The embodiments disclosed herein improve the accuracy of eye trackingdevices by compensating for common error-inducing factors such as, byway of non-limiting example, user head movements, head pressure, imageocclusion by cameras or trackers, and/or the independent image displaysto each eye. The embodiments described herein account for theseerror-inducing factors by using a model to more accurately predict the3D eye gaze location based on positional data obtained from one or moreeye trackers for each eye and the assumption of constant interpupillarydistance (and/or other eye tracking characteristics). In particular,each eye of the user is measured by its own, independent eye tracker.Those of skill in the art will realize that the eye tracking systemsdisclosed herein may be utilized in similar (e.g., non-teleoperational)applications benefiting from more accurate gaze-assisted system and/orinstrument control. By utilizing the eye tracking systems and methodsdisclosed herein, a user may experience more intuitive and moreefficient interaction with a teleoperational medical system.

According to various embodiments, minimally invasive medical proceduresmay be performed using a teleoperational system to guide instrumentdelivery and operation. Referring to FIG. 1A of the drawings, ateleoperational medical system for use in, for example, medicalprocedures including diagnostic, therapeutic, or surgical procedures, isgenerally indicated by the reference numeral 10. As will be described,the teleoperational medical systems of this disclosure are under theteleoperational control of a surgeon. In alternative embodiments, ateleoperational medical system may be under the partial control of acomputer programmed to perform the procedure or sub-procedure. In stillother alternative embodiments, a fully automated medical system, underthe full control of a computer programmed to perform the procedure orsub-procedure, may be used to perform procedures or sub-procedures. Asshown in FIG. 1 , the teleoperational medical system 10 generallyincludes a teleoperational assembly 12 near or mounted to an operatingtable O on which a patient P is positioned. The teleoperational assembly12 may be referred to as a patient-side manipulator (PSM). A medicalinstrument system 14 is operably coupled to the teleoperational assembly12. An operator input system 16 allows a surgeon or other type ofclinician S to view images of or representing the surgical site and tocontrol the operation of the medical instrument system 14. The operatorinput system 16 may be referred to as a master or surgeon's console. Oneexample of a teleoperational surgical system that can be used toimplement the systems and techniques described in this disclosure is ada Vinci® Surgical System manufactured by Intuitive Surgical, Inc. ofSunnyvale, Calif.

The teleoperational assembly 12 supports the medical instrument system14 and may include a kinematic structure of one or more non-servocontrolled links (e.g., one or more links that may be manuallypositioned and locked in place, generally referred to as a set-upstructure) and a teleoperational manipulator. (See, e.g., FIG. 2 ) Theteleoperational assembly 12 includes plurality of motors that driveinputs on the medical instrument system 14. These motors move inresponse to commands from a control system 22. The motors include drivesystems which when coupled to the medical instrument system 14 mayadvance the medical instrument into a naturally or surgically createdanatomical orifice. Other motorized drive systems may move the distalend of the medical instrument in multiple degrees of freedom, which mayinclude three degrees of linear motion (e.g., linear motion along the X,Y, Z Cartesian axes) and in three degrees of rotational motion (e.g.,rotation about the X, Y, Z Cartesian axes). Additionally, the motors canbe used to actuate an articulable end effector of the instrument.

The teleoperational medical system 10 also includes an image capturesystem 18 which includes an image capture device, such as an endoscope,and related image processing hardware and software. The teleoperationalmedical system 10 also includes a control system 22 that is operativelylinked to sensors, motors, actuators, and other components of theteleoperational assembly 12, the operator input system 16 and to theimage capture system 18.

The operator input system 16 may be located at a surgeon's console,which is usually located in the same room as operating table O. Itshould be understood, however, that the surgeon S can be located in adifferent room or a completely different building from the patient P.Operator input system 16 generally includes one or more controldevice(s) for controlling the medical instrument system 14. Morespecifically, in response to the surgeon's input commands, the controlsystem 22 effects servomechanical movement medical instrument system 14.The control device(s) may include one or more of any number of a varietyof input devices, such as hand grips, joysticks, trackballs, datagloves, trigger-guns, hand-operated controllers, foot-operatedcontrollers, voice recognition devices, touch screens, body motion orpresence sensors, and the like. In some embodiments, the controldevice(s) will be provided with the same degrees of freedom as themedical instruments of the teleoperational assembly to provide thesurgeon with telepresence, the perception that the control device(s) areintegral with the instruments so that the surgeon has a strong sense ofdirectly controlling instruments as if present at the surgical site. Inother embodiments, the control device(s) may have more or fewer degreesof freedom than the associated medical instruments and still provide thesurgeon with telepresence. In some embodiments, the control device(s)are manual input devices which move with six degrees of freedom, andwhich may also include an actuatable handle for actuating instruments(for example, for closing grasping jaws, applying an electricalpotential to an electrode, delivering a medicinal treatment, and thelike).

The system operator sees images, captured by the image capture system18, presented for viewing on a display system 20 operatively coupled toor incorporated into the operator input system 16. The display system 20displays an image or representation of the surgical site and medicalinstrument system(s) 14 generated by sub-systems of the image capturesystem 18. The display system 20 and the operator input system 16 may beoriented so the operator can control the medical instrument system 14and the operator input system 16 with the perception of telepresence.The display system 20 may include multiple displays such as separateright and left displays for presenting separate images to each eye ofthe operator, thus allowing the operator to view stereo images.

Alternatively or additionally, display system 20 may present images ofthe surgical site recorded and/or imaged preoperatively orintra-operatively using imaging technology such as computerizedtomography (CT), magnetic resonance imaging (MRI), fluoroscopy,thermography, ultrasound, optical coherence tomography (OCT), thermalimaging, impedance imaging, laser imaging, nanotube X-ray imaging, andthe like. The presented preoperative or intra-operative images mayinclude two-dimensional, three-dimensional, or four-dimensional(including e.g., time based or velocity based information) images andassociated image data sets for reproducing the images.

The control system 22 includes at least one memory and at least oneprocessor (not shown), and typically a plurality of processors, foreffecting control between the teleoperational system 12, medicalinstrument system 14, the operator input system 16, the image capturesystem 18, and the display system 20. The control system 22 alsoincludes programmed instructions (e.g., a computer-readable mediumstoring the instructions) to implement some or all of the methodsdescribed in accordance with aspects disclosed herein. While controlsystem 22 is shown as a single block in the simplified schematic of FIG.1 , the system may include two or more data processing circuits with oneportion of the processing optionally being performed on or adjacent theteleoperational assembly 12, another portion of the processing beingperformed at the operator input system 16, and the like. Any of a widevariety of centralized or distributed data processing architectures maybe employed. Similarly, the programmed instructions may be implementedas a number of separate programs or subroutines, or they may beintegrated into a number of other aspects of the teleoperational systemsdescribed herein. In one embodiment, control system 22 supports wirelesscommunication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11,DECT, and Wireless Telemetry.

In some embodiments, control system 22 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system 14. Responsive to the feedback, the servo controllerstransmit signals to the operator input system 16. The servocontroller(s) may also transmit signals instructing teleoperationalassembly 12 to move the medical instrument system(s) 14 which extendinto an internal surgical site within the patient body via openings inthe body. Any suitable conventional or specialized servo controller maybe used. A servo controller may be separate from, or integrated with,teleoperational assembly 12. In some embodiments, the servo controllerand teleoperational assembly are provided as part of a teleoperationalarm cart positioned adjacent to the patient's body.

In this embodiment, the teleoperational medical system 10 also includesan eye tracking unit 24 which may be operatively coupled to orincorporated into the operator input system 16. The eye tracking unit 24is operatively coupled to the control system 22 for sensing, measuring,recording, and conveying information related to the operator's eyeswhile the operator is viewing the display 20 and/or operating theoperator controls at the operator input system 16.

The teleoperational medical system 10 may further include optionaloperation and support systems (not shown) such as illumination systems,steering control systems, irrigation systems, and/or suction systems. Inalternative embodiments, the teleoperational system may include morethan one teleoperational assembly and/or more than one operator inputsystem. The exact number of manipulator assemblies will depend on thesurgical procedure and the space constraints within the operating room,among other factors. The operator input systems may be collocated orthey may be positioned in separate locations. Multiple operator inputsystems allow more than one operator to control one or more manipulatorassemblies in various combinations.

FIG. 1B is a front elevation view of a teleoperational assembly 100(e.g., the teleoperational assembly 12 shown in FIG. 1A) according toone embodiment. The assembly 100 includes a base 102 that rests on thefloor, a support tower 104 that is mounted on the base 102, and severalarms that support surgical tools (including portions of the imagecapture system 18). As shown in FIG. 1B, arms 106 a, 106 b, 106 c areinstrument arms that support and move the surgical instruments used tomanipulate tissue, and arm 108 is a camera arm that supports and movesthe endoscope. FIG. 1B further shows interchangeable surgicalinstruments 110 a, 110 b, 110 c mounted on the instrument arms 106 a,106 b, 106 c, respectively, and it shows an endoscope 112 mounted on thecamera arm 108. The endoscope 112 may be a stereo endoscope forcapturing stereo images of the surgical site and providing the separatestereo images to the display system 20. Knowledgeable persons willappreciate that the arms that support the instruments and the camera mayalso be supported by a base platform (fixed or moveable) mounted to aceiling or wall, or in some instances to another piece of equipment inthe operating room (e.g., the operating table). Likewise, they willappreciate that two or more separate bases may be used (e.g., one basesupporting each arm).

As is further illustrated in FIG. 1B, the instruments 110 a, 110 b, 110c, and the endoscope 112 include instrument interfaces 150 a, 150 b, 150c, and 150 d, respectively, and instrument shafts 152 a, 152 b, 152 c,and 152 d, respectively. In some embodiments, the teleoperationalassembly 100 may include supports for cannulas that fix the instruments110 a, 110 b, 110 c, and the endoscope 112 with respect to the cannulas.In some embodiments, portions of each of the instrument arms 106 a, 106b, 106 c, and 108 may be adjustable by personnel in the operating roomin order to position the instruments 110 a, 110 b, 110 c, and theendoscope 112 with respect to a patient. Other portions of the arms 106a, 106 b, 106 c, and 108 may be actuated and controlled by the operatorat an operator input system 120 (as shown in FIG. 1C). The surgicalinstruments 110 a, 110 b, 110 c, and endoscope 112, may also becontrolled by the operator at the operator input system 120.

FIG. 1C is a front elevation view of an operator input system 120 (e.g.,the operator input system 16 shown in FIG. 1A). The operator inputsystem 120 includes a console 121 equipped with left and right multipledegree-of-freedom (DOF) control interfaces 122 a and 122 b, which arekinematic chains that are used to control the surgical instruments 110a, 110 b, 110 c, and the endoscope 112. The surgeon grasps a pincherassembly 124 a, 124 b on each of control interfaces 122, typically withthe thumb and forefinger, and can move the pincher assembly to variouspositions and orientations. When a tool control mode is selected, eachof control interfaces 122 is configured to control a correspondingsurgical instrument and instrument arm 106. For example, a left controlinterface 122 a may be coupled to control the instrument arm 106 a andthe surgical instrument 110 a, and a right control interface 122 b maybe coupled to the control instrument arm 106 b and the surgicalinstrument 110 b. If the third instrument arm 106 c is used during asurgical procedure and is positioned on the left side, then left controlinterface 122 a can be switched from controlling the arm 106 a and thesurgical instrument 110 a to controlling the arm 106 c and the surgicalinstrument 110 c. Likewise, if the third instrument arm 106 c is usedduring a surgical procedure and is positioned on the right side, thenthe right control interface 122 a can be switched from controlling thearm 106 b and surgical instrument 110 b to controlling the arm 106 c andthe surgical instrument 110 c. In some instances, control assignmentsbetween the control interfaces 122 a, 122 b and combination of arm 106a/surgical instrument 110 a and combination of arm 106 b/surgicalinstrument 110 b may also be exchanged. This may be done, for example,if the endoscope is rolled 180 degrees, so that the instrument moving inthe endoscope's field of view appears to be on the same side as thecontrol interface the surgeon is moving. The pincher assembly istypically used to operate a jawed surgical end effector (e.g., scissors,grasping retractor, and the like) at the distal end of a surgicalinstrument 110.

Additional controls are provided with foot pedals 128. Each of footpedals 128 can activate certain functionality on the selected one ofinstruments 110. For example, foot pedals 128 can activate a drill or acautery tool or may operate irrigation, suction, or other functions.Multiple instruments can be activated by depressing multiple ones ofpedals 128. Certain functionality of instruments 110 may be activated byother controls.

The surgeon's console 120 also includes a stereo image viewer system 126(e.g., the display system 20 shown in FIG. 1A). Stereo image viewersystem 126 includes a left eyepiece 125 a and a right eyepiece 125 b, sothat the surgeon may view left and right stereo images using thesurgeon's left and right eyes respectively inside the stereo imageviewer system 126. Left side and right side images captured by endoscope112 are outputted on corresponding left and right image displays, whichthe surgeon perceives as a three-dimensional image on a display system(e.g., the display system 20 shown in FIG. 1A In an advantageousconfiguration, the control interfaces 122 are positioned below stereoimage viewer system 126 so that the images of the surgical tools shownin the display appear to be located near the surgeon's hands below thedisplay. This feature allows the surgeon to intuitively control thevarious surgical instruments in the three-dimensional display as ifwatching the hands directly. Accordingly, the servo control of theassociated instrument arm and instrument is based on the endoscopicimage reference frame.

The endoscopic image reference frame is also used if the controlinterfaces 122 are switched to a camera control mode. In some cases, ifthe camera control mode is selected, the surgeon may move the distal endof endoscope 112 by moving one or both of the control interfaces 122together. The surgeon may then intuitively move (e.g., pan, tilt, zoom)the displayed stereoscopic image by moving the control interfaces 122 asif holding the image in his or her hands.

As is further shown in FIG. 1C, a headrest 130 is positioned abovestereo image viewer system 126. As the surgeon is looking through stereoimage viewer system 126, the surgeon's forehead is positioned againstheadrest 130. In some embodiments of the present disclosure,manipulation of endoscope 112 or other surgical instruments can beachieved through manipulation of headrest 130 instead of utilization ofthe control interfaces 122. In some embodiments, the headrest 130 can,for example, include pressure sensors, a rocker plate, opticallymonitored slip plate, or other sensors that can detect movement of thesurgeon's head. Additional details on using a sensing method tomanipulate the headrest in order to control the endoscope camera may befound, for example, in U.S. application Ser. No. 61/865,996, entitled“ENDOSCOPE CONTROL SYSTEM,” which is incorporated herein by reference.

FIG. 1D is a front view of a vision cart component 140 of a surgicalsystem. For example, in one embodiment, the vision cart component 140 ispart of the medical system 10 shown in FIG. 1A. The vision cart 140 canhouse the surgical system's central electronic data processing unit 142(e.g., all or portions of control system 22 shown in FIG. 1A) and visionequipment 144 (e.g., portions of the image capture system 18 shown inFIG. 1A). The central electronic data processing unit 142 includes muchof the data processing used to operate the surgical system. In variousimplementations, however, the electronic data processing may bedistributed in the surgeon console 120 and teleoperational assembly 100.The vision equipment 144 may include camera control units for the leftand right image capture functions of the endoscope 112. The visionequipment 144 may also include illumination equipment (e.g., a Xenonlamp) that provides illumination for imaging the surgical site. As shownin FIG. 1D, the vision cart 140 includes an optional touch screenmonitor 146 (for example a 24-inch monitor), which may be mountedelsewhere, such as on the assembly 100 or on a patient side cart. Thevision cart 140 further includes space 148 for optional auxiliarysurgical equipment, such as electrosurgical units, insufflators, suctionirrigation instruments, or third-party cautery equipment. Theteleoperational assembly 100 and the surgeon's console 120 are coupled,for example via optical fiber communications links, to the vision cart140 so that the three components together act as a single teleoperatedminimally invasive surgical system that provides an intuitivetelepresence for the surgeon.

Note that in some embodiments, some or all of the assembly 100 of theteleoperated surgical system can be implemented in a virtual (simulated)environment, wherein some or all of the image seen by the surgeon at thesurgeon's console 120 can be synthetic images of instruments and/oranatomy. In some embodiments, such synthetic imagery can be provided bythe vision cart component 140 and/or directly generated at the surgeon'sconsole 120 (e.g., via a simulation module).

During a typical minimally invasive surgical procedure with theteleoperated surgical system described with reference to FIGS. 1A-1D, atleast two incisions are made into the patient's body (usually with theuse of a trocar to place the associated cannula). One incision is forthe endoscope camera instrument, and the other incisions are for thesurgical instruments. In some surgical procedures, several instrumentand/or camera ports are used to provide access and imaging for asurgical site. Although the incisions are relatively small in comparisonto larger incisions used for traditional open surgery, a minimum numberof incisions is desired to further reduce patient trauma and forimproved cosmesis. In other embodiments, the teleoperational medicalsystem 10 may be used with single incision access to the patient anatomyor with access through natural orifices such as the nose, mouth, anus,vagina, etc.

During a typical teleoperated surgery, it is often necessary for asurgeon to physically manipulate various controls to control thesurgical system, the imaging devices, and/or the other surgicalinstruments associated with the system. For example, a surgeon may needto adjust the field of view of the imaging device by physicallymanipulating controls to guide and influence the device. The surgeon mayuse his or her hand to manually control a joystick or mouse, or his orher foot to tap a foot pedal at the surgeon's console to log-in to thesurgical system, to search for a target surgical site within the view ofthe endoscope, to operate the movement of a surgical instrument such asa clamp, and/or to adjust the system settings or display settings. Theconventional methods require the surgeon to free one hand from surgicaloperation, or to use one foot to tap the foot pedal, both of which mayunnecessarily delay or disrupt the surgical operation. For example, thehand or foot action may redirect the surgeon's gaze and attention fromthe target surgical site to the surgeon's console, which could delay ordisrupt the operation. After performing the required manual adjustment,the surgeon may need to spend additional time refocusing his or herattention and gaze point on the target surgical site.

Embodiments disclosed herein utilize gaze detection to enhance the wayone or more users (e.g., surgeons and/or trainers) interface with thesurgical system By translating the user's eye gaze (e.g., the locationof a user's eye gaze relative to a display system or other surgicalsystem component) into commands directed to the surgical system,embodiments disclosed herein may enable faster and more efficientcontrol over the teleoperational medical system 10 than provided byconventional control methods. Eye tracking, or eye-gaze tracking, is theprocess of measuring either point-of-gaze (POG) (i.e., where the user islooking, typically in 3D space), or the motion of an eye relative to ahead. In other words, POG is the point in space where a person's gaze isdirected to, and has also been defined as the point in space that isimaged on the center of the highest acuity region of the retina (i.e.,the fovea) of each eye.

FIG. 2A illustrates a block diagram of a user U (e.g., a surgeon or aproctor) relative to an image display 151 (e.g., the image displaysystem 20 shown in FIG. 1A) and a surgical field 155. The user (and hisor her eyes) exists in a first 3D coordinate frame 160, the imagedisplay 151 includes a second 3D coordinate frame 165, and the surgicalfield exists in a third 3D coordinate frame 170. Each coordinate frame160, 165, 170 includes different dimensions and properties from theothers. As the user shifts his or her gaze in the first frame 160relative to the image display 151 in the second frame 165, theembodiments disclosed herein can translate that eye motion into acontrol signal to correspondingly influence the teleoperational medicalsystem 10 and/or a surgical instrument in the second frame 165 of thedisplay and/or the third frame 170 of the surgical field.

In one aspect, the eye-gaze tracking and observation of other eyecharacteristics can be used to communicate with and/or influence thebehavior of the teleoperational medical system 10 as a whole. Forexample, the eye characteristics and dynamics observed by the eyetracking unit 24 shown in FIG. 1A may be used for surgeon recognitionand log-in (e.g., in a manner similar to retinal scans). In someinstances, the eye gaze of the user can be used to better calibrate the3D positions of surgical instruments in space (e.g., in the thirdcoordinate frame 170) and account for the possible inaccuracies of thetelerobotic arm kinematic chain. In some embodiments, a user interface(e.g., a menu) may be overlaid upon the image of the surgical fieldshown on the image display. In some instances, the eye gaze of the userin the first coordinate frame 160 may be used to determine a viewinglocation on the image display 151 in the second coordinate frame 165,and can identify a user's selection among user selectable options of theuser interface corresponding to the determined viewing location. In someinstances, the 3D position of the user's gaze may be used to quantify ifthe user is seeing stereo or not based on the observed dynamics betweenthe two eyes.

In another aspect, real-time eye-gaze tracking can be used to activate,deactivate, and otherwise control distinct surgical instruments that arecoupled to the teleoperational medical system 10 such as, by way ofnon-limiting example, imaging devices and/or energy delivery devices.For example, the system 10 may be configured to activate a surgicalinstrument if the control system (e.g., a processor) determines that theviewing location matches the position of the surgical instrument for apredetermined length of time.

FIG. 2B illustrates a flowchart 180 describing an exemplary method ofusing the eye tracking unit to control and affect the teleoperationalmedical system 100 and/or any associated surgical instruments. Any ofthe method processes described herein may be implemented, at least inpart, in the form of executable code stored on non-transient, tangible,machine readable media that may be run by one or more processors. Atprocess 182, the user U, in the first coordinate frame 160 shown in FIG.2A, gazes at a particular 3D position in the image display 151, which isin the second coordinate frame 165. At process 184, the left and righteye trackers of the eye tracking unit (e.g., the eye tracking unit 24shown in FIG. 1A) observe and measure an eye characteristic (e.g., acharacteristic reflective of eye gaze) of the user U. In someembodiments, the eye trackers measure the eye gazes of each eye of theuser relative to the second coordinate frame 165 of the image display151. At process 186, the control system (e.g., the control system 22shown in FIG. 1A) uses the measured eye gaze data from the eye trackersto determine the 3D location on the image display 151 (within the secondcoordinate frame 165) at which the user's eyes are directed. In someembodiments, the control system may determine the viewed location bytracking incident angles of the light received by the eye trackers fromreflections off the eyes. In some embodiments, the processor 206 mayinitially perform a calibration process (e.g., the calibration process302 described in FIG. 4A) to determine baseline incident angles as theuser views target indicia that are displayed at known locations on theimage display 151, and generate a functional relationship between thedetected angles and the viewed locations on the image display 151. Thecontrol system can then track the incident angles as the user viewsother locations on the image display 151 and use the generatedfunctional relationship to determine (e.g., extrapolate from thecalibrated angles and locations) the corresponding viewed locations.

At process 188, the control system determines whether one of thedisplayed indicia (e.g., a menu option) on the image display 151 isbeing looked at by the user in a way that satisfies a defined conditionfor selection of that indicia. If so, at process 190, the user'sselection of the indicia causes the control system to initiate thefunction corresponding to the displayed indicia. For example, in someembodiments, the user's gaze may indicate the selection of an indiciaassociated with logging on to the teleoperational medical system 100, orwith the illumination of the image display 151, or with various othersystem settings.

If not, at process 192, the control system co-registers the viewed 3Dlocation in the second reference frame 165 to the corresponding 3Dlocation in the surgical field 155 in the third coordinate frame 170. Atprocess 194, the control system determines whether the user is lookingat the surgical field in a way that satisfies a defined condition formanipulating an imaging device or other surgical instrument. If so, atprocess 196, the user's gaze upon a particular area of the surgicalfield or a particular instrument within the surgical field causes thecontrol system to affect the relevant instrument in a fashioncorresponding to the characteristics of the user's gaze. For example, insome embodiments, as mentioned above, if the user gazes at a particularregion of the surgical field 155, the imaging device may “follow” theuser's gaze and re-center its field of view (e.g., to position thecenter of its field of view at the user's gaze point). In otherembodiments, if the user gazes at a particular surgical instrument for apredefined length of time, the surgical instrument may be activated. Ifnot, at process 198, the eye trackers continue to evaluate the user'sgaze for possible instructions.

There are a number of methods for measuring eye movement and gazedirection. In one method described herein, an infrared (IR) lightemitter emits IR light toward a user's eye. The IR light is reflectedfrom the user's retinas (through the pupils) back to an IR unit (e.g.,an IR camera or other imaging device), and the amount of reflected IRlight is based on the direction of the person's gaze relative to theemitter. In some embodiments, the user's gaze point in 3D space may bedetermined once the reflected IR light reaches a particular thresholdfor a certain amount of time. Small lapses in gaze can be interpreted asblinks and are typically ignored. Other eye tracking methods use videoimages from which the eye position is extracted, use search coils, orare based on electrooculograms.

FIG. 3A schematically illustrates an eye tracking system 220 formeasuring eye characteristics of the user such as eye position and eyemovement to determine his or her gaze point (e.g., “where a user islooking”) or the motion of the eye relative to the head. The eyetracking system 200 comprises an image display 208, at least one lightemitter 210, an eye tracking unit 212, and an optical assembly 213. Oneor more eye tracking systems 200 can be integrated in surgeon's console120. The optical assembly 213 is positioned in the light transmissionpath between the eye 202 and the light emitter 210, the eye trackingdetector 212, and the image display 208. The optical assembly 213directs eye tracking light (e.g., IR light) from the light emitter 210to the eye 202, the visible light from the image display 208 to the eye202, and the reflected eye tracking light (e.g., the IR light) from theeye 202 to the eye tracking detector 212. In this embodiment, theemitted eye tracking light, the reflected eye tracking light, and thevisible light share the optical path between the eye 202 and the opticalassembly 213. As described in greater detail in FIGS. 3B, 3C, and 3D,the eye tracking unit may be a stereo imaging device employing two ormore cameras or other imaging devices or employing a singlestereo-capture imaging device. Each eye tracking system 200 can be usedto independently track either the surgeon's left eye or right eye. Forexample, if the optical assembly 213 is used to direct light to thesurgeon's left eye (e.g., an eye 202 a shown in FIG. 3B), the eyetracking system 200 is configured to detect characteristics indicativeof left eye gaze. If the optical assembly 213 is used to direct light tothe surgeon's right eye (e.g., the eye 202 b shown in FIG. 3B), the eyetracking system 200 is configured to detect characteristics indicativeof right eye gaze. As described in greater detail in FIG. 3B, the eyetracking system 200 can be coupled with one or more processors forprocessing the eye characteristic data (e.g., pupillary position data orcorneal reflection data) of the eye 202 tracked by the eye trackingdetector 212.

As shown in FIG. 3A, visible light 211 a emitted from the image display208 is directed by the optical assembly 213 towards the surgeon's eye202. The eye tracking light (e.g. IR light) 211 b emitted by the lightemitter 210 is directed by the optical assembly 213 towards thesurgeon's eye, and is reflected by the surgeon's eye 202 back towardsthe optical assembly 213. The optical assembly 213 directs the reflectedeye tracking light from the surgeon's eye 202 towards the eye trackingunit 212.

In some embodiments, the optical assembly 213 includes one or moremirrors arranged to reflect both the visible light and the eye trackinglight. In alternative embodiments, the optical assembly 213 may includea beam splitter or other optical filtering element that reflects somelight beams while transmitting others. For example as described in FIG.3C, the optical assembly can include a dichroic element (e.g., a filteror a mirror) configured to selectively pass light in a particularwavelength range, such as IR light, while reflecting the light outsidethat particular wavelength range, such as visible light. The opticalassembly 213 may also include any other suitable number and arrangementof mirrors and/or other optical devices, such as, by way of non-limitingexample, a dichroic mirror (e.g., a partially reflecting mirror that ispartially reflective and partially transparent), a reflector having adichroic optical coating, a dichroic mirrored prism, birefringentmaterials, and/or polarizing beam splitters. The optical assembly 213allows the components within the eye tracking system 200 to have agreater variety of possible arrangements because the mirrors and/orfilters of the optical assembly 213 may effectively “hide” the eyetracking unit 212 from the surgeon's sight even though the light 211 bfrom the eye tracking unit 212 at least partially shares the sameoptical path as the visible light from the image display 208.

FIG. 3B is a diagram illustrating an eye tracking system 220 accordingto one embodiment of the present disclosure. The eye tracking system 220may be used by the teleoperational medical system 10 of FIGS. 1A, 1B,and 1C. For example, the eye tracking system 220 can be partially orwholly included in the stereo viewer 126 at the surgeon's console 120according to some embodiments of the present disclosure. The eyetracking system 220 includes left and right eyepieces 125 a and 125 b,the optical assembly 235 comprising left and right eye mirror sets 204 aand 204 b, left and right eye image displays 208 a and 208 b, left andright eye light emitters 210 a and 210 b, and left and right eyetracking units 212 a and 212 b. In the pictured embodiment, the eyetracking system 220 includes a left eye processor 214 a, a right eyeprocessor 214 b, and an integrator 216.

The left and right eyepieces 125 a, 125 b of the system 220 may becomponents of the surgeon's console 120 (see FIG. 1C). The left andright eyepieces 125 a, 125 b include lenses, and the surgeon may viewthe left and right image displays 208 a, 208 b through the left andright eyepieces 125 a, 125 b with the surgeon's left and right eyes,respectively. The lenses may focus light from (e.g., emitted orreflected from) a light source (e.g., the light emitters 210 a, 210 band/or the image displays 208 a, 208 b) towards a detector (such as theeyes 202 a, 202 b). The lenses may include objective lens that gatherslight from the target and focuses the light beam to produce a realimage. In some embodiments, the distance between the left and right eyepieces 125 a, 125 b are adjustable to accommodate differentinterpupillary distances of different users. In some embodiments, theleft and right eye pieces 125 a, 125 b may be adjusted independentlybased on the need of the surgeon's left and right eye vision,respectively. In some embodiments, the left and right eye pieces 125 a,125 b may include suitable optical coatings configured to minimizereflection and maximize transmission of light from the light emittersand/or left and right eye image displays 208 a, 208 b.

The left eye and right eye light emitters 210 a, 210 b emit light 211 bto illuminate the surgeon's left eye and right eye, respectively, andthe reflected light can be captured by the left and right eye trackingunits 212 a, 212 b, respectively, to track gaze points for the left eyeand right eye, respectively. In some embodiments, the left and rightlight emitters 210 a and 210 b may be infrared (IR) light emitters, suchas IR light emitting diodes (IR LEDs). In some embodiments, there may bemore than one light emitter for each eye (e.g., two light emitters foreach eye). The multiple light emitters for each eye may be spaced apartby a predetermined distance so that the light emitted from each lightemitter appears as separate reflections in a single eye. In someembodiments, the one or more left eye light emitters 210 a may beintegrated together with the left eye tracking units 212 a, and the oneor more right eye light emitters 210 b may be integrated together withthe right eye tracking units 212 b. Various embodiments may include anynumber of tracking units for each eye. Some embodiments may include anunequal number of tracking units for the left eye and the right eye.

The left eye tracking units 212 a can be used for tracking the gazepoint of the surgeon's left eye, and the right eye tracking units 212 bmay be used for tracking the gaze point of the surgeon's right eye. Asshown in FIG. 3B, each of the eye tracking units 212 a, 212 b is athree-dimensional imaging system including two eye tracking cameras forstereo eye-tracking. Each eye tracking unit may be used for trackingpupil position and corneal reflection for a respective eye of thesurgeon. For example, in the pictured embodiment, two left eye trackingcamera 212 a 1, 212 a 2 are used for tracking the pupil's position andcorneal reflection of the surgeon's left eye. Similarly, in the picturedembodiment, two right eye tracking camera 212 b 1, 212 b 2 are used fortracking the pupil's position and corneal reflection of the surgeon'sright eye. The multiple eye tracking units for each eye are positionedto be spaced apart from each other by a predetermined distance so thatstereo images of each pupil's position and corneal reflection may beindependently tracked by the more than one eye tracking unit, and a 3Dlocation of each pupil's position and corneal reflection may becalculated based on the collected data from the independent eye trackingunits for each eye. In alternative embodiments, each of the eye trackingunits may include a single imaging device, including for example, astereo camera, or may include more than two imaging devices.

In some embodiments, eye tracking units 212 a and 212 b are chargedcoupled device (CCD) cameras. In some embodiments, the eye trackingunits 212 a and 212 b are IR cameras that are sensitive to IR light andcan capture the infrared light emitted from IR light emitters. In someembodiments, the eye tracking units 212 a and 212 b may include a highzoom lens to provide images having higher magnification of the eyes(e.g., the pupils).

In some embodiments, the eye tracking units 212 a, 212 b and the lightemitters 210 a, 210 b are placed at the base of left eye and right eyeimage displays 208 a, 208 b, respectively. In some embodiments, the eyetracking units 212 a, 212 b may be located in the stereo image viewersystem 126 shown in FIG. 1C. The light emitters 210 a and 210 b may beintegrated together with the left and right eye tracking units 212 a and212 b, respectively. Typically, the user will position his or her leftand right eyes to directly face the left eye and right eye imagedisplays 125 a, 125 b, respectively. Thus, with this arrangement, eacheye tracking unit 212 a, 212 b is positioned to directly face the eye tobe tracked. In particular, the left and right eye tracking units 212 a,212 b are positioned to directly face the left eye and the right eye,respectively. The configuration disclosed herein may improve theconvenience and accuracy of the eye tracking process because thisconfiguration eliminates the need for external tracking devices, such aseye glasses. In addition, as described above, conventional eye gazetracking devices may be located near the eyepieces, thereby creatinginterference when the surgeon is looking into the eyepieces. Forexample, the edges of the eye gaze tracking device may appear in thesurgeon's vision, potentially distracting the surgeon or partiallyobscuring his view of the surgical field. Thus, the currentconfiguration may improve the surgeon's experience in using the eyetracking technology by eliminating any unnecessary interference imageswithin the surgeon's vision.

In the pictured embodiment and with reference to FIG. 2A, the processors214 a, 214 b are coupled to the left and right eye tracking units 212 a,212 b, respectively, and are configured to calculate the 3D location ofthe surgeon's gaze point with respect to the second coordinate frame 165of the image display 151 (e.g., displays 208 a, 208 b) and translatethat 3D position into the corresponding 3D position of the thirdcoordinate system 170 of the surgical field 155 (shown in FIG. 2A). Forexample, the gaze points captured by the left and right eye trackingunits 212 a, 212 b can be rectified, and the disparity between the gazepoints of the surgeon's left and right eyes can be determined. The 3Dlocation of the surgeon's gaze point can then be calculated using thedistance between the left and right eye tracking units 212 a, 212 b, theparameters related to the focal length of each of the left and right eyetracking units 212 a, 212 b, and the determined disparity.

A 3D stereo image of the surgical field may be perceived by the surgeonvia the eye tracking system 220. In some embodiments, the endoscope 112located at the teleoperational assembly 100 can be manipulated tocapture images of the surgical field 155 during a surgery (shown in FIG.2A), and these images can be shown on the left and right image displays208 a, 208 b. In some embodiments, the image displays 208 a, 208 b arethe same as the image display 151 shown in FIG. 2A. In some embodiments,the endoscope 112 comprises a stereoscopic camera. The images capturedby the endoscope 112 may then be processed by the processors 214 a, 214b to generate left and right stereo images, respectively. The generatedleft and right stereo images may be shown on left and right imagedisplays 208 a and 208 b, respectively. In particular, the left eyeimage display 208 a is communicatively coupled to the endoscope 112, andis configured to display a left eye stereo image of a surgical siteduring a surgery. Similarly, the right eye image display 208 b iscoupled to stereoscopic camera 112 and is configured to display a righteye stereo image of the surgical site. The left eye and right eye stereoimages are captured by stereoscopic camera 112 and processed for leftand right eye vision respectively. The left eye and right eye imagedisplays 208 a, 208 b may be 2D or 3D display screens. In someembodiments, the left eye and right-eye image displays 208 a, 208 b areliquid crystal display (LCD) screens. In some instances, the imagedisplays 208 a, 208 b may be presented simultaneously on multipledisplay screens or devices (e.g., an external screen or mobile device).

As mentioned above, the eye tracking system 220 includes a left eyeprocessor 214 a, a right eye processor 214 b, and an integrator 216. Theprocessors 214 a, 214 b and the integrator 216 are configured to processgaze point data received from the eye tracking units 212 a, 212 b todetermine a viewing location on the image displays 208 a, 208 b at whichthe gaze point of the user is directed, and to control at least onefunction of the teleoperational medical system 10 based on thedetermined viewing location. In particular, the processors 214 a, 214 bmay process pupillary position and corneal reflection point datareceived by left eye and right eye tracking units 212 a, 212 b. In someembodiments, the pupillary position and corneal reflection point datareceived by each eye tracking unit may be processed by processors 214 aand 214 b to determine the 3D location of the pupil's position andcorneal reflection. The integrator 216 may be used to integrate thepupillary position and corneal reflection data received from each eye toform a 3D gaze point or location of the surgeon during a surgery.

In some embodiments, the functions of the left eye and right eyeprocessors (and/or the integrator) may performed by a single processor.In some embodiments, the integrator 216 integrates the informationreceived by both processors 214 a, 214 b to determine and process theeye gaze locations of the user. In some embodiments, the processors 214a, 214 b and/or the integrator 216 may be located elsewhere within theteleoperational system 10 (e.g., within the vision cart 140 as part ofthe central electronic data processing unit 142, at the teleoperationalassembly 100, and/or within the surgeon's console 120). In someembodiments, the processors 214 a, 214 b and/or the integrator 216 canalso be coupled to a memory to store the gaze point measurement,registration, and calibration data. In some embodiments, the processors214 a, 214 b and/or the integrator 216 may be used to calculate the 2Dlocation of the surgeon's gaze point. As described in further detailbelow, in some embodiments, the head motion of the surgeon may becompensated for when determining the 3D location of the pupil's positionand corneal reflection.

In some embodiments, the calculated 2D or 3D location of the surgeon'sgaze point can be displayed in any of a variety of suitablerepresentations, such as dots, flags, or vectors showing the changes ofthe surgeon's gaze point. The surgeon's gaze point can be displayed incombination with the image of the surgical field 155 on the left andright image displays 208 a, 208 b. In some embodiments, the eye trackingsystem 220 may also be used in the surgeon's console 120 integrated witha simulation module, e.g., a da Vinci® Skills Simulator™, where virtualimages can be shown on the left and right image displays 208 a and 208b.

The optical assembly 235 is arranged relative to the eyes 202, the lightemitters 210, the eye tracking units 212, and the image displays 208 todirect the IR light from the light emitters 210 to the eyes 202, thevisible light from the image displays 208 to the eyes 202, and thereflected IR light (emitted from the light emitters 210) from the eyes202 to the eye tracking units 212. In particular, each of the left andright mirror sets 204 a, 204 b of the optical assembly 235 comprises aplurality of mirrors arranged to reflect the IR light from the left andright light emitters 210 a, 210 b, respectively, into the left and righteyes of the surgeon, respectively, and to reflect IR light from the leftand right eyes into the left and right eye tracking units, 212 a, 212 b,respectively.

Thus, the left eye mirror set 204 a includes a set of mirrors that arearranged to provide optical communication between the surgeon's lefteye, the left eye tracking units 212 a, and the left eye light emitter210 a. For example, the mirrors 204 a may be arranged, as shown in FIG.3B, so that the left eye portion of the stereo image shown on the lefteye image display 208 a can be directed via visible light 211 a into thesurgeon's left eye to be seen by the surgeon. The left eye mirror set204 a may also allow the IR light 211 b emitted from left eye lightemitter 210 a to be directed onto the surgeon's left eye. The reflectedIR light 211 b from the surgeon's left eye may be directed by left eyemirror set 204 a to the left eye tracking units 212 a, thereby enablingthe left eye tracking unit 212 a to track the reflected IR light todetermine the 3D location of the gaze point of the surgeon's left eye.

Similarly, the right eye mirror set 204 b includes a set of mirrors thatare arranged to provide optical communication between the surgeon'sright eye, the right eye tracking units 212 b, and the right eye lightemitter 210 b. For example, the mirrors 204 b may be arranged, as shownin FIG. 3B, so that the right eye portion of the stereo image shown onthe right eye image display 208 b can be directed with visible light 211a to the surgeon's right eye to be seen by the surgeon. The right eyemirror set 204 b may also allow the IR light 211 b emitted from righteye light emitter 210 b to be directed onto the surgeon's right eye. Thereflected IR light 211 b from the surgeon's right eye may be directed bythe right eye mirror set 204 b to right eye tracking units 212 b,thereby enabling the right eye tracking unit 212 b to track thereflected IR light to determine the 3D location of the gaze point of thesurgeon's right eye. In FIG. 3B, the path of the emitted and reflectedIR light 211 b shares an optical path with the visible light 211 abetween the display 208 a and the eye 202 a and between the display 208b and the eye 202 b.

As mentioned above, in some embodiments, the left eye and right eyetracking units 212 a, 212 b may also be used to track 3D head motions ofthe surgeon (e.g., in the first coordinate frame 160 shown in FIG. 2A).In some embodiments, the tracking units 212 a, 212 b track the 3D motionof the surgeon's head by monitoring a fixed reference point on thesurgeon's head (e.g., including the face). Because the reference pointdoes not have continuously synchronized motion with the surgeon's eyegaze point, the reference point may be used to estimate the surgeon'shead position and orientation. The reference point may comprise any of avariety of anatomical landmarks or features on the surgeon's head. Forexample, the reference point may include a head feature such as eyelidcurvatures, corners of the eyes, irises, and/or eyebrows. In someexamples, the head feature that is used as the reference point may besegmented and tracked. For example, in one embodiment, the corner of theeye may be detected and located by the intersection of two splines ofthe eye. When two stereo units (e.g., the left eye and right eyetracking units 212 a, 212 b) are used, the head feature may be trackedin 3D space. The interpupillary distance between the surgeon's left andright eyes is assumed to be constant, and used by the processor as aconstant factor in its calculations. When the surgeon moves his or herhead or face without shifting his or her gaze point, the surgeon'spupils remain stationary relative to the eye tracking units 212 a, 212b. Accordingly, the reference point on the head moves while thesurgeon's pupils remain fixed relative to the eye tracking units 212 a,212 b. Therefore, there is a relative motion between the left and rightpupils and the reference point on the face/head. This relative motionmay be tracked and monitored by the left eye and right eye trackingunits 212 a and 212 b to calculate appropriate compensation values.These compensation values may be used to compensate for the tracked 3Dhead motions in determining the 3D location of the gaze point of each ofthe surgeon's eyes.

FIG. 3C is a diagram illustrating an eye tracking system 250 includingan optical assembly 255 that operates as a beam splitter. The eyetracking system 250 is an exemplary embodiment of the eye trackingsystem 200 of FIG. 3A. The eye tracking system 250 can be partially orwholly included in the stereo image viewer system 126 on the surgeon'sconsole 120 according to various embodiments of the present disclosure.The eye tracking system 250 includes the left eye tracking units 212 a(with eye tracking cameras 212 a 1, 212 a 2), the right eye trackingunits 212 b (with eye tracking cameras 212 b 1, 212 b 2), the left eyelight emitter 210 a, the right eye light emitter 210 b, the left eyeimage display 208 a, the right eye image display 208 b, the left eyeprocessor 214 a, the left eye processor 214 b, and the integrator 216.However, in the eye tracking system 250 shown in FIG. 3C, thesecomponents are arranged in a different configuration than in the eyetracking system 220. In the eye tracking system 220 shown in FIG. 3B,the eye tracking units 212 a, 212 b and the light emitters 210 a, 210 bare located beneath the image displays 208 a, 208 b (relative to theuser). The optical assembly 235 is positioned between the image displays208 a, 208 b and the eyepieces 125 a, 125 b. In contrast, in the eyetracking system 250 shown in FIG. 3C, the left eye and right eyetracking units 212 a, 212 b are located in front of the image displays208 a, 208 b in an area or space between the left eye and right eyeimage displays 208 a, 208 b and the optical assembly 255. The opticalassembly 255 comprises the left eye mirror set 204 a and the right eyemirror set 204 b. The left eye mirror set 204 a comprises a firstoptical element 256 and a second optical element 257. The right eyemirror set 204 b comprises a third optical element 258 and a fourthoptical element 259. The optical elements 256, 257, 258, and 259 can beany of a variety of light-transmitting and/or light-reflecting opticaldevices, including without limitation, mirrors, filters, and prisms.Although two processors 214 a, 214 b are shown in FIG. 3C, one withordinary skill in the art would understand that any number of processorsmay be arranged in any suitable topology.

In some embodiments, the optical assembly 255 includes at least onemirror set including a beam splitter, such as, by way of nonlimitingexample, a dichroic mirror or a dichroic filter. The beam splitter maycomprise any device capable of both transmission and reflection of lightwithin distinct wavelength ranges. For example, as shown in FIG. 3C, thesecond optical element 257 of the left eye mirror set 204 a includes aleft beam splitter 257 that allows IR light 211 b to pass through whilereflecting visible light 211 a, such as the visible light emitted fromthe left eye stereo image display 208 a. Thus, the IR light 211 bemitted from the left eye light emitter 210 a can pass through the leftbeam splitter 257 to illuminate the left eye, and the reflected IR light112 b from the surgeon's left eye 202 a can also pass through the leftbeam splitter 257 to be captured by left eye tracking unit 212 a. At thesame time, the visible light 211 a emitted from left eye image display208 a can be directed by the optical elements 256 and 257 of mirror set204 a to be seen by the surgeon's left eye 202 a. In some embodiments,the left beam splitter 257 comprises a dichroic mirror.

Similarly, the third optical element 258 of the right eye mirror set 204b includes a right beam splitter that allows IR light 211 b to passthrough while reflecting visible light 211 a, such as the visible lightemitted from the right eye stereo image display 208 b. Thus, the IRlight 211 b emitted from the right eye light emitter 210 b can passthrough the right beam splitter 258 to illuminate the right eye, and thereflected IR light 211 b from the surgeon's right eye 202 b can alsopass through the right beam splitter 258 to be captured by right eyetracking unit 212 b. At the same time, the visible light 211 a emittedfrom right eye image display 208 b can be directed by the mirror set 204b (and the beam splitter 258) to be seen by the surgeon's right eye 202b. In some embodiments, the right beam splitter 258 comprises a dichroicmirror.

The inclusion of beam splitters 257, 258 in the optical assembly 255allows for a configuration in which the eye tracking units 212 a, 212 bshare at least part of the optical path for the displayed imagesoriginating from the image displays 208 a, 208 b without being visibleto the surgeon (e.g., in the image displays 208 a, 208 b). In otherwords, the reflected IR light 211 b shares a portion of the optical pathof visible light 211 a, namely the portion of the optical path betweenthe eye and the respective beam splitter (e.g., between eye 202 a andbeam splitter 257 and between eye 202 b and beam splitter 258). As shownin FIG. 3C, the left eye and right eye tracking units 212 a, 212 b arepositioned behind the beam splitters 203 a, 203 b and directly facingthe surgeon's eyes. This configuration allows the eye tracking units 212a, 212 b to capture information from the surgeon's left and right eyes202 a, 202 b, respectively, without obstructing the visible light pathand without having any optical reflection paths affect the imagedisplays 208 a, 208 b (e.g., without the image displays 208 a, 208 breceiving any reflected IR light from the surgeon's eyes). In contrast,as shown in FIG. 3B, the reflected IR light (shown by dashed linearrows) from the surgeon's eyes 202 a, 202 b is reflected back towardthe image displays 208 a, 208 b to be received by the eye tracking units212 a, 212 b.

In some instances, the incorporation of a beam splitter (e.g., the beamsplitter 257, 258) in the optical assembly 255 of the eye trackingsystem 250 allows the light 211 b from eye tracking units 212 a, 212 bto share at least part of the optical path of the visible light 211 afrom the displayed images without having the eye tracking units 212 a,212 b visible to the surgeon. In some instances, the surgeon may be lessdistracted by interference images and more focused on the currentprocedure because this configuration may eliminate interference imagesfrom the eye tracking units in the surgeon's image displays 208 a, 208b. In addition, the configuration shown in FIG. 3C may also provide moreflexibility or compactness for the design and manufacturing of the eyetracking system 250. The configuration of system 250 may create aclearer view of the display (without the eye tracking unit visible)while accommodating minimized space and design constraints.

FIG. 3D is a diagram illustrating an exemplary eye tracking system 280including an optical assembly 260 that operates as a beam splitteraccording to one embodiment of the present disclosure. The eye trackingsystem 280 is another exemplary embodiment of the eye tracking system200 shown in FIG. 3A. The eye tracking system 280 may be partially orwholly included in stereo image viewer system 126 on the surgeon'sconsole 120 according to some embodiments of the present disclosure. Theeye tracking system 280 includes the left eye tracking units 212 a (witheye tracking cameras 212 a 1, 212 a 2), the right eye tracking units 212b (with eye tracking cameras 212 b 1, 212 b 2), the left eye lightemitter 210 a, the right eye light emitter 210 b, the left eye imagedisplay 208 a, the right eye image display 208 b, the left eye processor214 a, the left eye processor 214 b, and the integrator 216. In the eyetracking system 280 shown in FIG. 3D, these components are arranged in adifferent configuration than in the eye tracking system 250. Inparticular, in the eye tracking system 250 shown in FIG. 3C, the eyetracking units 212 a, 212 b and the light emitters 210 a, 210 b arepositioned between the image displays 208 a, 208 b and the opticalassembly 255. In contrast, in the eye tracking system 280 shown in FIG.3D, the eye tracking units 212 a, 212 b and the light emitters 210 a,210 b are located adjacent to (e.g., lateral to) the optical assembly260 (or rotated approximately 90° with respect to the eyes). In thepictured embodiment of FIG. 3D, the space between the optical assembly260 and the image displays 208 a, 208 b is unoccupied, thereby providingan unobstructed optical path between the image displays 208 a, 208 b andthe optical assembly 260.

Like the optical assembly 255 shown in FIG. 3C, the optical assembly 260of the eye tracking system 280 includes two optical elements that act asbeam splitters. In the optical assembly 260 shown in FIG. 3D, the firstand fourth optical elements 256 and 259 act as beam splitters. Inparticular, the first optical element 256 of the left eye mirror set 204a includes a left beam splitter, and the fourth optical element 9 of theright eye mirror set 204 b includes a right beam splitter. As shown inFIG. 3D, the left beam splitter 256 of the left eye mirror set 204 atransmits IR light 211 b through while reflecting visible light 211 a,such as the visible light emitted from the left eye stereo image display208 a. Thus, the IR light 211 b emitted from the left eye light emitter210 a can pass through the left beam splitter 256 to illuminate the lefteye, and the reflected IR light from the surgeon's left eye 202 a canalso pass through the left beam splitter 256 to be captured by left eyetracking unit 212 a. At the same time, the visible light emitted fromleft eye image display 208 a can be directed by the mirror set 204 a(including the beam splitter 256) to be seen by the surgeon's left eye202 a. In some embodiments, the left beam splitter 6 comprises adichroic mirror.

Similarly, the right beam splitter 259 of the right eye mirror set 204 btransmits IR light 211 b through while reflecting visible light 211 a,such as the visible light emitted from the right eye stereo imagedisplay 208 b. Thus, the IR light 211 b emitted from the right eye lightemitter 210 b can pass through the right beam splitter 259 to illuminatethe right eye, and the reflected IR light 211 b from the surgeon's righteye 202 b can also pass through the right beam splitter 259 to becaptured by right eye tracking unit 212 b. At the same time, the visiblelight 211 a emitted from right eye image display 208 b can be directedby the mirror set 204 b (and the beam splitter 259) to be seen by thesurgeon's right eye 202 b. In some embodiments, the right beam splitter259 comprises a dichroic mirror.

The inclusion of the beam splitters 257, 259 in the optical assembly 260allows for a configuration in which the light 211 b from eye trackingunits 212 a, 212 b share at least part of the optical path for thedisplayed images originating from the image displays 208 a, 208 bwithout the eye tracking units being visible to the surgeon. Inparticular, as shown in FIG. 3D, the left eye and right eye trackingunits 212 a, 212 b are positioned behind the beam splitters 257, 259 andat an angle (e.g., 90°) to the surgeon's eyes. This configuration allowsthe eye tracking units 212 a, 212 b to capture information from thesurgeon's left and right eyes, respectively, without obstructing thepath of the visible light from the displays and without having anyoptical reflection paths affect the image displays 208 a, 208 b (e.g.,without the image displays 208 a, 208 b receiving any reflected IR lightfrom the surgeon's eyes).

In some instances, the incorporation of a beam splitter (e.g., the beamsplitter 256, in the optical assembly 260 of the eye tracking system 280allows the light 211 b from eye tracking units 212 a, 212 b to share atleast part of the optical path of the visible light 211 a from thedisplayed images without having the eye tracking units 212 a, 212 bvisible to the surgeon. In some instances, the surgeon may be lessdistracted by interference images and more focused on the currentprocedure because this configuration may eliminate interference imagesfrom the eye tracking units in the surgeon's image displays 208 a, 208b. In addition, the configuration shown in FIG. 3D may also provide moreflexibility or compactness for the design and manufacturing of the eyetracking system 280. The configuration of system 280 may create aclearer view of the display (without the eye tracking unit visible)while accommodating minimized space and design constraints.

It is to be understood that the position of the light emitters 210(e.g., left eye and right eye light emitters 210 a and 210 b) isflexible. The position of the eye tracking units 212 (e.g., the left andright eye tracking units 212 a and 212 b) is also flexible. One ofordinary skill in the art would understand that the light emitter 212and/or the eye tracking unit 212 can be located at any suitable positionrelative to the surgeon and the image displays (e.g., on the surgeon'sconsole 120) to minimize interference to the surgeon's vision and toimprove the accuracy and efficiency of the eye tracking process.

FIG. 4A illustrates a method 300 for determining the surgeon's 3D gazepoint using the eye tracking systems as shown in FIGS. 3A-3D accordingto an embodiment of the present disclosure. The method 300 includes twoprocesses: a calibration process 302 and a measurement process 304. Insome embodiments, the calibration process 302 is a 3D calibrationprocess, where the surgeon's gaze point in the 3D space is compared witha predetermined target in the 3D space with known 3D locationparameters.

The image of the target T shown in the 3D space may be separated intoleft and right stereo images, and displayed on the left and right imagedisplays 208 a, 208 b, respectively. The left eye and right eye lightemitters 210 a, 210 b may emit light that can be tracked by the left eyeand right eye tracking units 212 a, 212 b, respectively. The left eyeand right eye mirror sets 204 a and 204 b may be arranged so that theleft eye and right eye stereo images of the target displayed on the lefteye and right eye image displays 208 a and 208 b can be reflected anddirected into the surgeon's left and right eyes 202 a, 202 b,respectively. In some embodiments, as shown in FIGS. 3A-3D, the left eyeand right eye mirror sets 204 a and 204 b are arranged so that there isno “crosstalk” or shared optical pathways between the left eye and righteye optical communication. Light, such as IR light 211 b, emitted fromthe light emitters 210 a, 210 b can also be directed through the lefteye and right eye mirror sets 204 a, 204 b to illuminate the surgeon'sleft and right eyes 202 a, 202 b. During the calibration process, the 3Dlocation of the target is predetermined, for example, with known 3Dlocation parameters in the 3D space, so that the measured data may becompared with the predetermined location data of the target subject todetermine the compensation values and/or gaze point determination modelin the following process.

FIG. 4B illustrates an example of the left eye tracking units 212 atracking the corneal reflection and the pupillary location of thesurgeon's left eye 202 a when a predetermined target T (having the 3Dcoordinates aT, bT, cT) is shown on the left eye image display 208 aduring the calibration process. The calibration process 302 may begin atprocess 312 by showing a target Tin the 3D space. As shown in FIG. 4B,the target T can be placed in a predetermined 3D coordinate frame 350with coordinate values of (aT, bT, cT) on the left eye image display 208a. The target T shown on the left eye image display 208 a may be a lefteye stereo image captured by stereoscopic unit 112 and processed byprocessor 214 a. In other embodiments, the target T may be a virtuallycreated image. The left eye mirror set 204 a (not shown) can be situatedat any suitable position relative to the left eye eye tracking units 212a and the surgeon's left eye 202 a, as discussed above in relation toFIGS. 3A-3D. It is to be understood that although only one cornealreflection of the surgeon's left eye is shown in FIG. 4B for the sake ofsimplicity, there may be more than one corneal reflection from the morethan one light emitters 210 for each eye 202.

In some examples, the target T may be a surgical tool icon shown in the3D space 350. The target T may also be a moving target, or a target thatmay change size dynamically. Alternatively, the target may also be anactual surgical tool in the surgical field, the location of which can betracked and identified using any suitable tool tracking technology. Forexample, the calibration process 302 may incorporate features disclosedin U.S. Patent Publication No. 2006/0258938, entitled “Methods andsystem for performing 3D tool tracking by fusion of sensor and/or cameraderived data during minimally invasive robotic surgery,” filed on May16, 2005, which is incorporated herein by reference. During thecalibration process 302, the 3D location of the target T ispredetermined, for example with known 3D location parameters in the 3Dspace (e.g., the coordinate values of (aT, bT, cT)), so that themeasured data may be compared with the known location parameters of thetarget T to determine various models in the following steps.

In the pictured embodiment, the calibration process 302 proceeds toprocess 314 by receiving the pupil's 2D position and 2D cornealreflection data of left and right eyes captured by the left and righteye tracking units 212 a and 212 b, respectively. In some embodiments,the 2D pupillary position and 2D corneal reflection data may includecoordinate values, displacements, and/or angles. In some embodiments,left eye and right eye mirror sets 204 a and 204 b are arranged as shownin FIGS. 3A-3D, so that the corneal reflections of the surgeon's leftand right eyes 202 a, 202 b can be directed into the left eye and righteye tracking units 212 a, 212 b, respectively, to be tracked. Thetracked 2D data may then be received and processed by the left eye andright eye processors 214 a, 214 b and/or the integrator 216 to obtainthe 3D pupillary position and corneal reflection data of surgeon's leftand right eyes 202 a, 202 b.

FIG. 4C illustrates a 2D coordinate frame 360 corresponding to thesurgeon's cornea, and a 2D coordinate frame 370 corresponding to thesurgeon's pupil. As shown in FIG. 4C, when the left eye light emitter210 a emits light toward the surgeon's eye 202 a, the light (e.g., IRlight) reflects off the surface of a cornea of the surgeon's left eye202 a, and the reflection image becomes a bright region with a cornealreflection center 365 (having the coordinates (uR, vR)) placed in thepredetermined 2D coordinate frame 360. In some instances, the cornealreflection data includes tracking information for more than one cornealreflection per eye. The center of the pupil 375 (having the coordinates(xP, yP)) in the predetermined 2D coordinate frame 370 can also betracked by the left eye tracking units 212 a.

The calibration process 302 proceeds to process 316 by tracking the headmotion to more accurately determine the pupillary position and cornealreflection data of left and right eyes, respectively. In other words,the head motion data may be used to compensate for the headmotion-induced inaccuracies of the pupillary position and cornealreflection data. For example, rotations of the head can be approximatedby changes in the head motion data. In some embodiments, as describedabove in relation to FIG. 3A, a head feature (e.g., a corner of an eye)of the surgeon may be tracked to determine the head motion. In someembodiments, the surgeon may be requested to perform one or more headmotions while focusing on a target T in the image displays 208 a, 208 bto determine the appropriate compensation values and/or compensationmodel. For example, the surgeon may be requested to move his or her headcloser to the eyepieces 125 a, 125 b while focusing on a predeterminedtarget T on the image displays 208 a, 208 b. The surgeon may also berequested to move his or her head in a series of motions (e.g., up anddown, left and right, in a rotation, and/or further away from theeyepieces) to gather more calibration information.

The left eye and right eye tracking units 212 a and 212 b may capturethe relative motion between the surgeon's pupils and the face/headfeature to determine the compensation values related to the face/headmotion. In some embodiments, the head motion of the surgeon may also betracked by one or more sensors mounted on the headrest 130. Thecalculated 2D or 3D gaze point location may be further adjusted orcompensated for based on the tracked head motion of the surgeon. Thedata collected by the sensors may be combined with the data acquired atprocess 316 to determine the compensation values. Additional details onusing the sensing method to manipulate the headrest may be found, forexample, in U.S. Application No. 61/865,996, entitled “ENDOSCOPE CONTROLSYSTEM.”

In some embodiments, the compensation values and/or compensation modelmay be saved in a memory coupled to the processors 214 a, 214 b and/orthe integrator 216 for future measurements or calculations. In someembodiments, the compensation values and/or compensation model of aspecific user may be saved in the memory as part of the user's profiledata so that the same user does not need to repeat the calibrationprocess when the user logs into the system again.

The calibration process 302 proceeds to process 318 by determining the3D pupillary position and 3D corneal reflection data of each of thesurgeon's left and right eyes 202 a, 202 b, respectively. In someembodiments, the eye tracking units 212 a include stereo cameras, andstereo images including the 2D pupillary position and 2D cornealreflection data can be captured and processed by the processors 214 a tocalculate a disparity between the multiple stereo images. For example,as shown in FIG. 4B, when the one or more left eye tracking units 212 ainclude a stereo camera that can capture stereo images of the surgeon'sleft eye 202 a, the 2D pupillary position 375 (xP, yP) and 2D cornealreflection data 365 (uR, vR) from each individual left eye tracking unit212 a may be processed to calculate a disparity between the multiplestereo images. In some embodiments, the determined position data mayinclude the 2D pupillary position and 2D corneal reflection data of thesurgeon's left eye 202 a.

The determined 2D data of each eye may then be combined to estimate the3D eye gaze location of the surgeon. In some embodiments, the determinedposition data may include the pupil's 3D position and 3D cornealreflection data. A disparity to depth conversion map may be obtainedduring the calibration process using this method. The 3D data includingdepth of the surgeon's pupil (coordinate point zP) and depth of cornealreflection (coordinate point wR) may then be estimated using thedisparity. For example, the 3D eye gaze data of the surgeon's left eye202 a may be calculated using the distance between between—the left eyetracking units 212 a, the parameters related to the focal length of eachof the left eye tracking units 212 a, and the calculated disparity. Adisparity to depth conversion map may be obtained during the calibrationprocess using this method. In some embodiments, the surgeon's headmotion may also be captured by the left and right eye trackers 204 a and204 b.

In some embodiments, the surgeon's pupil may show a vergence (e.g., asindicated by an angle γ shown in FIG. 4B) when looking at a target Twith different depths. The depth of the surgeon's pupil and cornealreflection may be estimated by monitoring the vergence (e.g., the angleγ) of each of the surgeon's pupils and forming a conversion chartbetween the measured interpupillary distance and the depth of the targetT in the 3D coordinate frame 350. In some embodiments, the depth of thesurgeon's pupil (e.g., zP) and depth of corneal reflection (e.g., wR)may be calculated using a triangulation method as illustrated in FIG.4B. For example, the distance between the multiple left eye trackingunits 212 a is set to be a distance dC (e.g., the distance betweenpoints A and B in FIG. 4B), and the triangulation angles α and β may betracked when the surgeon's eye is looking at target T at differentlocations in 3D coordinate frame 350. In one instance, when the target Tis located at different depths, the angles α and β may changeaccordingly, so that a distance dP (e.g., the distance between point Cand the pupil) between the eye tracking units 212 a and the pupil can becalculated. The distance value dP can be further used to calculate the3D pupillary position (e.g., coordinates (xP, yP, zP)). The 3D pupillaryposition (e.g., coordinates (xP, yP, zP)) and corneal reflection data(e.g., coordinates (uR, vR, wR)) of the surgeon's right eye may bedetermined in a substantially similar manner as the determination of thesurgeon's left eye.

At process 322 of the calibration process 302, the determined 3Dpupillary position and 3D corneal reflection data is compared with thepredetermined 3D location parameters of the predetermined target T todetermine a gaze point determination model. In other words, the gazepoint determination model can be formed by determining the relationshipbetween the 3D pupillary position and 3D corneal reflection data and the3D location of the predetermined target T. In some examples, thedetermined 3D pupillary position 375′ (coordinates (xP, yP, zP)) and the3D corneal reflection data 365′ (coordinates ((uR, vR, wR)) of thesurgeon's left eye 202 a is compared with the 3D location data T(coordinates (aT, bT, cT)) of the left stereo image of the target Tchosen at process 312 to obtain the following relationship or functionf:(aT,bT,cT)=f(xP,yP,zP,uR,vR,wR)

In some embodiments, a plurality of calibration targets are used for thecalibration processes, and the parameters of the function f may bedetermined using the pupil's position and corneal reflection datagathered from the plurality of target points during the calibrationprocess. In some examples, methodologies such as least squaresoptimization, or maximum likelihood estimation may be used to determinethe parameters of the function f. In some embodiments, a gaze directionvector for each eye can be formed using the 3D pupillary position and 3Dcorneal reflection data of each eye and an intersection of each gazedirection vector may be determined to be the surgeon's gaze point. Thedetermined gaze point may then be compared with the 3D location data ofthe target T to determine the function f. In some embodiments during thecalibration process, the error between the 3D location calculated usingfunction f and the actual predetermined location of the target T may beminimized using an optimization method, such as least squaresoptimization, maximum likelihood estimation. In some embodiments, thegaze point determination model may also be formed using 2D position dataof the surgeon's eyes, the 2D location data of the target, and thevergence (e.g., angle γ) of the surgeon's pupils. In some embodiments,the gaze point determination model may also include a matrix showing theconversion from the pupil's 3D position and 3D corneal reflection datato the 3D location of the target T in a coordination system in the 3Dspace.

Similarly, the 3D pupillary position and 3D corneal reflection data ofthe surgeon's right eye 202 b may be compared with the 3D location dataof the right stereo image of the target T chosen at process 312.

In some embodiments, the calibration process 302 may be repeatedmultiple times, so that the accuracy of the gaze point determinationmodel may be improved until the accuracy satisfies a predeterminedsystem requirement. In some embodiments, after a first gaze pointdetermination model (e.g., function f) is formed, one or more realtargets may be used to estimate the accuracy of the first gaze pointdetermination model. For example, by rerunning the mapping optimizationusing the real target(s), the first gaze point determination model maybe updated to form a second gaze point determination model. The accuracybetween the first and second models is compared and evaluated, so that amore accurate gaze point determination model may be formed.

After the calibration process 302 is completed, the method 300 proceedsto a measurement process 304. The measurement process 304 may be carriedout during a surgery or a training process when the endoscope orstereoscopic camera 112 is capturing an image of a surgical site. Insome embodiments, the calibration process 302 and the measurementprocess 304 may also be conducted in a simulated exercise using asimulation module, for example using a da Vinci® Skills Simulator™(e.g., that may be integrated with the surgeon's console 120).

The measurement process 304 starts at process 324 by receiving the 2Dpupillary position and 2D corneal reflection data of surgeon's left andright eyes 202 a, 202 b, respectively, when the surgeon is looking atimages (e.g., a surgical site or virtual image) displayed on left eyeand right eye image displays 208 a, 208 b. The configuration and methodfor process 324 may be substantially similar to process 314 of method300 as previously discussed. In some embodiments, the image of thesurgical site may be captured by the endoscope or stereoscopic camera112 and processed to be separated into left eye and right eye stereoimages displayed on the left eye and right eye image displays 208 a, 208b, respectively. The 2D pupillary position and 2D corneal reflectiondata of surgeon's left and right eyes are captured by the left eye andright eye tracking units 212 a, 212 b, respectively, and are processedto obtain the 3D pupillary position data by the processors 214 a, 214 band/or the integrator 216.

The measurement process 304 proceeds to process 326 by estimating thepupillary position and corneal reflection data for each of the surgeon'sleft and right eyes 202 a, 202 b using the compensation values and/orcompensation model determined at process 316. In some embodiments, thecaptured heard/face motion at step 316 may also be used to compensatethe pupil's position and corneal reflection data or the surgeon's 3Dgaze point. As described above, the head motion data may be trackedduring the calibration process 302 by tracking a head feature using theleft eye and right eye tracking units 212 a, 212 b, and the compensationvalues and/or compensation model may be used to calculate the change tothe pupillary position and/or the corneal reflection data induced by thehead motion. The calculated change value may be used to adjust thepupillary position and corneal reflection data for the surgeon's leftand right eyes determined at process 324 (e.g. to compensate for thesurgeon's head motion). In particular, a function between the headmotion and the tracked motions of the eye corners can be formed duringthe calibration process 302. During the measurement process 304, themotions of the head/eye feature may also be tracked, and the surgeon'shead motions may be estimated using the formed function from thecalibration process 302. The 3D pupillary position and 3D cornealreflection data may then be converted to the surgeon's 3D gaze pointlocation by the processors 214 by using the gaze point determinationmodel obtained at step 322.

The measurement process 304 proceeds to process 328 by determining the3D pupillary position and 3D corneal reflection data of each of thesurgeon's left and right eyes 202 a, 202 b. The process for determiningthe 3D pupillary position and 3D corneal reflection data may besubstantially similar to process 318 of method 300 as previouslydiscussed. In some embodiments, the 2D pupillary position and 2D cornealreflection data received from each of the left eye tracking units 212 amay be processed by the left eye processor 214 a. The pupil's positionand corneal reflection of left eye 202 a can then be calculated usingthe relative position between corneal reflection center 365 of thecorneal reflection, and the center of the pupil 375 (shown in FIG. 4C).The 3D pupillary position and 3D corneal reflection data of thesurgeon's right eye may be determined in a substantially similar manneras the determination of the surgeon's left eye. For example, the 2Dpupillary position and 2D corneal reflection data received from each ofthe right eye tracking units 212 b may be processed by the right eyeprocessor 214 b. The pupil's position and corneal reflection of righteye 202 b can then be calculated using the relative position betweencorneal reflection center of the right corneal reflection, and thecenter of the right pupil.

The measurement process 304 proceeds to process 330 by determining the3D gaze point of the surgeon using the gaze point determination modelobtained at process 322 of the calibration process 302. In someembodiments, the 3D pupillary position and 3D corneal reflection data ofthe surgeon's both eyes determined at process 328 may be processed usingthe gaze point determination model to determine the 3D location of thesurgeon's gaze point. In some examples, the gaze direction vector foreach eye can be formed using the determined 3D pupillary position and 3Dcorneal reflection data of each eye at process 328. The intersection ofeach gaze direction vector may then be used to determine the surgeon'sgaze point.

After the measurement process 304, in some embodiments, the determined3D gaze point location may be shown onto the image displays 208 shown inFIGS. 3A-3D. The 3D gaze point location may be expressed in any of avariety of suitable representations, such as, without limitation, dots,lines, vectors, arrows, and semi-transparent circles. The gaze pointmeasured by the teleoperational medical system 10 as discussed above maybe used in various applications, including both real-time medicalprocedures and virtual training procedures.

In an exemplary aspect, an first eye tracking system comprises an imagedisplay configured to display an image of a surgical field to a user; aright eye tracker configured to measure data about a first gaze point ofa right eye of the user, the right eye tracker including a right stereoimaging device; a left eye tracker configured to measure data about asecond gaze point of a left eye of the user, the left eye trackerincluding a left stereo imaging device; and at least one processorconfigured to process the data about the first gaze point and the secondgaze point to determine a viewing location in the displayed image atwhich the gaze point of the user is directed. In another exemplaryaspect, the right stereo imaging device of the first eye tracking systemincludes at least two cameras configured to receive light from the righteye and the left stereo imaging device includes at least two camerasconfigured to receive light from the left eye. In another exemplaryaspect, the at least one processor of the first eye tracking system isconfigured to process the data about the first gaze point and the secondgaze point to determine the viewing location based on a constant factorcorresponding to a constant interpupillary distance.

In another exemplary aspect, the right eye tracker of the first eyetracking system is configured to detect the 2D corneal reflection dataof the right eye, and the left eye tracker is configured to detect the2D corneal reflection data of the left eye.

In another exemplary aspect, the right eye tracker of the first eyetracking system is configured to detect the 2D pupillary position dataof the right eye, and the left eye tracker is configured to detect the2D pupillary position data of the left eye.

In another exemplary aspect, the right eye tracker and the left eyetracker of the first eye tracking system are configured to trackpositional data about a fixed reference point corresponding to a headfeature of the user.

In another exemplary aspect, the at least one processor of the first eyetracking system is configured to process the data about the first gazepoint and the second gaze point and compensate for head motions of theuser to determine the viewing location based on the positional dataabout the fixed reference point.

In another exemplary aspect, the first eye tracking system furtherincludes a right eye light emitter and a left eye light emitter, theright eye light emitter configured to emit light of a first wavelengthrange to the right eye of the user, and the left eye light emitterconfigured to emit light of the first wavelength range to the left eyeof the user.

In another exemplary aspect, the first eye tracking system furtherincludes an optical assembly positioned between the image display andthe eyes of the user, the optical assembly comprising a right eye mirrorset and a left eye mirror set arranged to provide optical communicationbetween the eyes of the user, the eye trackers, and the light emitters.

In another exemplary aspect, the right eye mirror set of the first eyetracking system is configured to direct light of a second wavelengthrange from the image display to the right eye of the user, to direct thelight of the first wavelength range from the right eye light emitter tothe right eye of the user, and to direct reflected light of the firstwavelength range from the right eye of the user to the right eyetracker, and the left eye mirror set is configured to direct light of asecond wavelength range from the image display to the left eye of theuser, to direct the light of the first wavelength range from the lefteye light emitter to the left eye of the user, and to direct reflectedlight of the first wavelength range from the left eye of the user to theleft eye tracker.

In another exemplary aspect, the optical assembly of the first eyetracking system is configured to reflect the light of the secondwavelength range from the image display and to transmit the light of thefirst wavelength range from the light emitters.

In another exemplary aspect, the right eye mirror set of the first eyetracking system includes a right beamsplitter configured to reflectlight of a second wavelength range from the image display to the righteye of the user, to transmit the light of the first wavelength rangefrom the right eye light emitter to the right eye of the user, and totransmit reflected light of the first wavelength range from the righteye of the user to the right eye tracker, and the left eye mirror setincludes a left beamsplitter configured to reflect light of a secondwavelength range from the image display to the left eye of the user, totransmit the light of the first wavelength range from the left eye lightemitter to the left eye of the user, and to transmit reflected light ofthe first wavelength range from the left eye of the user to the left eyetracker.

In another exemplary aspect, the right eye light emitter and the righteye tracker of the first eye tracking system are disposed between theright beamsplitter and the image display, and the left eye light emitterand the left eye tracker are disposed between the left beamsplitter andthe image display.

In another exemplary aspect, the left eye light emitter and the left eyetracker of the first eye tracking system are disposed lateral to theleft beamsplitter and in a plane between the image display and the lefteye, and the right eye light emitter and the right eye tracker aredisposed lateral to the right beamsplitter and in a plane between theimage display and the right eye.

In an exemplary aspect, a first teleoperational medical system forperforming a medical procedure, comprises an eye tracking system thatincludes an image display configured to display an image of a surgicalfield to a user; at least one right eye tracker configured to measuredata about a first gaze point of a right eye of the user; at least oneleft eye tracker configured to measure data about a second gaze point ofa left eye of the user; and at least one processor configured to processthe data about the first gaze point and the second gaze point todetermine a viewing location in the displayed image at which the gazepoint of the user is directed; and a control unit configured to controlat least one function of the teleoperational medical system based uponthe determined viewing location.

In another exemplary aspect, the image display of the firstteleoperational medical system is a 3D image display configured todisplay to the user a 3D image of the surgical field.

In another exemplary aspect, the first teleoperational medical systemfurther comprises a surgical instrument, wherein the control unit isconfigured to control at least one function of the surgical instrumentin the surgical field based upon the determined viewing location in thedisplayed image.

In another exemplary aspect, the image display of the firstteleoperational medical system is configured to display to the user animage of a user interface comprising a plurality of functional options.

In another exemplary aspect, the control unit of the firstteleoperational medical system is configured to initiate at least one ofthe plurality of functional options if the determined viewing locationmatches a position of the at least one functional option in thedisplayed image of the user interface.

In another exemplary aspect, the at least one processor of the firstteleoperational medical system is configured to process the data aboutthe first gaze point and the second gaze point to determine the viewinglocation based on a constant factor corresponding to a constantinterpupillary distance.

In another exemplary aspect, the at least one right eye tracker of thefirst teleoperational medical system is configured to detect the 2Dcorneal reflection data of the right eye, and the at least one left eyetracker is configured to detect the 2D corneal reflection data of theleft eye.

In another exemplary aspect, the at least one right eye tracker of thefirst teleoperational medical system is configured to detect the 2Dpupillary position data of the right eye, and the at least one left eyetracker is configured to detect the 2D pupillary position data of theleft eye.

In another exemplary aspect, the at least one right eye tracker and theat least one left eye tracker of the first teleoperational medicalsystem are configured to track positional data about a fixed referencepoint corresponding to a head feature of the user.

In another exemplary aspect, the at least one processor of the firstteleoperational medical system is configured to process the data aboutthe first gaze point and the second gaze point and compensate for headmotions of the user to determine the viewing location based on thepositional data about the fixed reference point.

In another exemplary aspect, the first teleoperational medical systemfurther includes a right eye light emitter and a left eye light emitter,the right eye light emitter configured to emit light of a firstwavelength range to the right eye of the user, and the left eye lightemitter configured to emit light of a first wavelength range to the lefteye of the user.

In another exemplary aspect, the first teleoperational medical systemfurther includes an optical assembly positioned between the imagedisplay and the eyes of the user, the optical assembly comprising aright eye mirror set and a left eye mirror set arranged to provideoptical communication between the eyes of the user, the eye trackers,and the light emitters.

In another exemplary aspect, the right eye mirror set of the firstteleoperational medical system is configured to direct light of a secondwavelength range from the image display to the right eye of the user, todirect the light of the first wavelength range from the right eye lightemitter to the right eye of the user, and to direct reflected light ofthe first wavelength range from the right eye of the user to the atleast one right eye tracker, and the left eye mirror set is configuredto direct light of a second wavelength range from the image display tothe left eye of the user, to direct the light of the first wavelengthrange from the left eye light emitter to the left eye of the user, andto direct reflected light of the first wavelength range from the lefteye of the user to the at least one left eye tracker.

In another exemplary aspect, the optical assembly of the firstteleoperational medical system is configured to reflect the light of thesecond wavelength range from the image display and to transmit the lightof the first wavelength range from the light emitters.

In another exemplary aspect, the right eye mirror set of the firstteleoperational medical system includes a right beamsplitter configuredto reflect light of a second wavelength range from the image display tothe right eye of the user, to transmit the light of the first wavelengthrange from the right eye light emitter to the right eye of the user, andto transmit reflected light of the first wavelength range from the righteye of the user to the at least one right eye tracker, and the left eyemirror set includes a left beamsplitter configured to reflect light of asecond wavelength range from the image display to the left eye of theuser, to transmit the light of the first wavelength range from the lefteye light emitter to the left eye of the user, and to transmit reflectedlight of the first wavelength range from the left eye of the user to theat least one left eye tracker.

In another exemplary aspect, the right eye light emitter and the atleast one right eye tracker of the first teleoperational medical systemare disposed in front of the image display and between the rightbeamsplitter and the image display, and the left eye light emitter andthe at least one left eye tracker are disposed in front of the imagedisplay and between the left beamsplitter and the image display.

In another exemplary aspect, the left eye light emitter and the at leastone left eye tracker of the first teleoperational medical system aredisposed lateral to the left beamsplitter and in front of the imagedisplay, and the right eye light emitter and the at least one right eyetracker are disposed lateral to the right beamsplitter and in front ofthe image display.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of thedisclosure should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

What is claimed is:
 1. An eye tracking system, comprising: an image display configured to display a stereo image of a surgical field to a user, wherein the image display comprises a first coordinate frame and the surgical field comprises a second coordinate frame and wherein the user is in a third coordinate frame; a right eye tracker configured to measure data about a first gaze point of a right eye of the user; a left eye tracker configured to measure data about a second gaze point of a left eye of the user; and at least one processor configured to process the data about the first gaze point and the second gaze point to determine a viewing location in the displayed stereo image at which a three-dimensional gaze point of the user is directed.
 2. The eye tracking system of claim 1, wherein the at least one processor is configured to process the data about the first gaze point and the second gaze point to determine the viewing location based on a constant factor corresponding to a constant interpupillary distance.
 3. The eye tracking system of claim 1, wherein the right eye tracker includes at least two cameras configured to receive light from the right eye and the left eye tracker includes at least two cameras configured to receive light from the left eye.
 4. The eye tracking system of claim 1, wherein the right eye tracker is configured to detect 2D corneal reflection data of the right eye, and the left eye tracker is configured to detect 2D corneal reflection data of the left eye.
 5. The eye tracking system of claim 1, wherein the right eye tracker is configured to detect a 2D pupillary position data of the right eye, and the left eye tracker is configured to detect a 2D pupillary position data of the left eye.
 6. The eye tracking system of claim 1, wherein the right eye tracker is configured to monitor a vergence of a pupil of the right eye and the left eye tracker is configured to monitor a vergence of a pupil of the left eye.
 7. The eye tracking system of claim 1, wherein the at least one processor is configured to determine a right gaze direction vector from the data about the first gaze point and to determine a left gaze direction vector from the data about the second gaze point.
 8. The eye tracking system of claim 7, wherein three-dimensional gaze point is determined from an intersection of the right gaze direction vector and the left gaze direction vector.
 9. The eye tracking system of claim 1, wherein the at least one processor is configured to display a graphical representation on the image display of the viewing location at which a three-dimensional gaze point of the user is directed.
 10. The eye tracking system of claim 1, wherein the at least one processor is configured to determine a disparity between the first and second gaze point to determine the viewing location in the displayed stereo image at which a three-dimensional gaze point of the user is directed.
 11. The eye tracking system of claim 1, wherein the stereo image includes a surgical field captured by a stereo endoscopic imaging system.
 12. The eye tracking system of claim 1, wherein the stereo image includes a surgical field generated by a simulation module.
 13. The eye tracking system of claim 1, wherein the at least one processor is further configured to determine the three-dimensional gaze point of the user within the first coordinate frame.
 14. The eye tracking system of claim 1, wherein the at least one processor is further configured to determine the three-dimensional gaze point of the user within the second coordinate frame.
 15. The eye tracking system of claim 1, wherein the right eye tracker and the left eye tracker are configured to track positional data about a fixed reference point corresponding to a head feature of the user.
 16. The eye tracking system of claim 15, wherein the at least one processor is configured to process the data about the first gaze point and the second gaze point and compensate for head motions of the user to determine the viewing location based on the positional data about the fixed reference point.
 17. The eye tracking system of claim 1, further including a right eye light emitter and a left eye light emitter, the right eye light emitter configured to emit light of a first wavelength range to the right eye of the user, and the left eye light emitter configured to emit light of the first wavelength range to the left eye of the user.
 18. The eye tracking system of claim 17, further including an optical assembly positioned between the image display and the eyes of the user, the optical assembly comprising a right eye mirror set and a left eye mirror set arranged to provide optical communication between the eyes of the user, the eye trackers, and the light emitters.
 19. The eye tracking system of claim 18, wherein the right eye mirror set is configured to direct light of a second wavelength range from the image display to the right eye of the user, to direct the light of the first wavelength range from the right eye light emitter to the right eye of the user, and to direct reflected light of the first wavelength range from the right eye of the user to the right eye tracker, and the left eye mirror set is configured to direct light of a second wavelength range from the image display to the left eye of the user, to direct the light of the first wavelength range from the left eye light emitter to the left eye of the user, and to direct reflected light of the first wavelength range from the left eye of the user to the left eye tracker. 